The Journal of Educational Research & Policy Studies 
February 2013 Vol. 1, pp. 50-66 


Bolstering the Impact of Online Professional Development for Teachers 

Todd D. Reeves 

Boston College 

Joseph J. Pedulla 

Boston College 

Abstract 

Online professional development (OPD) for teachers is an increasingly popular and viable 
alternative to face-to-face professional development. While OPD can be effective, little is known 
about OPD’s design and implementation features that maximize its impact. Using data from a 
large-scale OPD initiative, this correlational study (N = 1231) investigates antecedents of self- 
reported changes in teacher knowledge, classroom practice, and student achievement. Three 
regression analyses replicate the importance of several factors in effective professional 
development, or online learning more generally, and also identify additional predictors of OPD’s 
impact(s). The paper also discusses an applied framework for conceptualizing and modeling the 
effects of OPD’s features on its successive outcomes. Implications for the design, 
implementation and evaluation of OPD, directions for future research, and study limitations are 
discussed. 
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Introduction 

Recently, online professional development (OPD) has proliferated in an effort to eliminate 
various barriers to face-to-face professional development, e.g. access and scheduling. However, 
since OPD is a recent innovation (Dede, Ketelhut, Whitehouse, Breit, & McCloskey, 2009), 
those hoping to design and implement it have available little in the way of objective, research- 
based knowledge to guide their efforts. Indeed, there is a paucity of research conducted 
systematically to leam how to bolster OPD’s impact on teachers and students (Ginsberg, Gray, & 
Levin, 2004; O’Dwyer et ah, 2010). Instead, the bulk of the literature around OPD is more 
anecdotal or suggestive than it is empirically based (Whitehouse, Breit, McCloskey, Ketelhut, & 
Dede, 2006). 

In their recent research agenda for OPD, Dede et al. (2009) discussed the problems associated 
with the current state of the OPD literature: 

Until more rigorous [online teacher professional development] research is conducted, 
developers are hard pressed to know the best design features to include, educators remain 
uninformed about which programs will help support teacher change and student learning, 
and funders lack sufficient guidelines for where to direct their support (p. 8). 

Hence, this present study addressed the research question of how to increase the impact of OPD. 
More specifically, we attempt to link particular OPD design and implementation features to 
OPD’s impact on teachers’ knowledge, classroom practices, and their students’ achievement. 
Such research-based knowledge can be incorporated into the design and implementation of 
future OPD programs for teachers and can also provide more of an empirical basis for the 
training of OPD facilitators (Russell, Kleiman, Carey, & Douglas, 2009). 

Conceptual Framework and Literature Review 

Desimone’s (2009) conceptual framework for professional development activities contends that: 

• Professional development activities change participants' knowledge and skills and/or 
attitudes or beliefs. 

• Participants implement the new knowledge, skills, attitudes or beliefs to improve classroom 
practice. 

• Instructional change in turn fosters increased student achievement. 

During professional development impact studies, Desimone recommends measuring these 
intennediate and ultimate outcomes, and also advises identifying variables that mediate and 
moderate such effects. With this study, we focus on three such outcomes, and also attempt to 
understand OPD factors that might hinder or promote these effects. 

Our study draws primarily on two extant bodies of research, namely the literatures on the 
effectiveness of face-to-face professional development and online learning. These two bodies of 
relevant literature both frame our study and provide a context for the interpretation of the 
findings. Regarding the effectiveness of professional development, Desimone’s (2009) recent 
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review of the literature showed that effective professional development is characterized by: (a) a 
focus on subject matter content, (b) extended duration, (c) coherence, (d) opportunities for active 
learning, and (e) collective participation. 


Lessons about OPD can also be learned from the broader online learning effectiveness literature. 
Earlier meta-analyses and literature reviews in this field have highlighted the importance of, for 
example: instructor involvement, interaction among participants, participants’ computer 
proficiency, time on task, learner activity and reflection, and feedback from the instructor (Zhao, 
Lei, Yan, Lai, & Tan, 2005; Tallent-Runnels et ah, 2006; Means, Toyama, Murphy, Bakia, & 
Jones, 2009). 

In contrast to the professional development and online learning effectiveness literatures, various 
scholars have noted a dearth of empirical research that could be utilized to design and implement 
effective OPD for teachers (e.g., Russell et ah, 2009). The field has learned through rigorous 
experimental research, however, that OPD can be effective. Across four randomized controlled 
trials in different grades/subjects, O’Dwyer et al. (2010) found medium-to-large OPD effects on 
teacher knowledge and instructional practices as well as some small effects on short-range 
student outcomes. 

To our knowledge, there are only two published studies that examined the impact of specific 
design/implementation features on OPD’s effectiveness. First, Carey, Kleiman, Russell, Venable 
and Louis (2008) showed no benefits of facilitation and peer interaction during a small-scale 
experimental study. Second, in a similar experimental study Russell et al. (2009) found no 
differences in OPD’s impact when manipulating the amount of supportive interactions provided 
by content experts, online facilitators and peers. 

The Present Study 

Since we have strong evidence for OPD’s effectiveness (e.g., O’Dwyer et al., 2010), the next 
logical step is to work to improve it. Indeed, the small, inconsistent student outcomes observed 
by O’Dwyer et al. imply that there is some room for improvement in OPD’s effect on in this 
area. This study, then, examines the effectiveness of OPD for teachers vis-a-vis its design and 
implementation features. For this study, we first attempt to build on findings reviewed earlier 
with the following rationale: It could be assumed that research-based knowledge from the 
disparate face-to-face PD and (more general) online learning literatures can be extrapolated to 
OPD. However empirical evidence which li nk s these design and implementation features directly 
to OPD effectiveness is preferable. In particular, we attempt to replicate findings regarding 
linking computer proficiency, school support, content focus, coherence, and an array of active 
learning variables to OPD effectiveness. Given that other literature on the effectiveness of OPD 
is scant, another contribution of this paper is the investigation of the effects of fourteen other 
variables not examined previously in the OPD literature (see Tables 1 and 2). 
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Our approach to studying OPD is in accordance with the principle that research on the 
effectiveness of an intervention should be coherent with the design of that intervention. In the 
case of teacher professional development (inclusive of OPD), the mechanism by which it 
theoretically operates to effect educational change is as follows: (a) teachers participate in PD 
(which varies along myriad design and implementation dimensions), (b) this participation 
changes their knowledge, skills, beliefs or attitudes, (c) their classroom practices then change 
accordingly, and finally (d) student outcomes improve (e.g., Desimone, 2009). Thus, the present 
study investigates relationships between OPD design and implementation features and each of 
three successive OPD outcomes: teachers’ self-reported levels of knowledge, classroom practice, 
and their students’ achievement — all aspects of various PD evaluation frameworks (e.g., 
Desimone, 2009; Guskey, 2000; Ketelhut, McCloskey, Dede, Breit, & Whitehouse, 2006). 

Our approach affords not only the ability to discover which, if any, factors matter, but also where 
in this chain of events each factor has its effect. As an example, school culture might not affect 
what teachers learn during OPD, but might instead influence their later implementation of that 
knowledge in the classroom. Alternatively, particular student characteristics might not influence 
teacher learning during professional development but could moderate OPD’s impact on student 
achievement. Examining variation in design and implementation features with respect to multiple 
outcomes should provide a nuanced understanding of the effectiveness of OPD. 

In addition to examining the effects of various design or implementation features, we also seek 
empirical evidence for this chain of successive outcomes. This is done by modeling teacher 
knowledge after the course as a predictor of improvement in classroom practice, and both teacher 
knowledge after the course and improvement in classroom practice as predictors of improvement 
in student achievement. Notably, including these variables as mediators also allows us to 
examine the effects of various factors on OPD’s “later” outcomes (e.g., improved classroom 
practice) above and beyond — or distinct from — the effects of its “earlier” outcomes (e.g., teacher 
knowledge gains). Aside from our substantive findings, another contribution of this paper is that 
it provides an example of the utility of this approach to modeling the effectiveness of OPD. 

Our data source for the present study was the e-Leaming for Educators (EfE) Initiative. The EfE 
initiative is a U.S. Department of Education-funded project implemented over the past five years 
in ten states. EfE is predicated on the belief that scheduling and access represent critical barriers 
to teachers’ participation in “high-quality” professional development. The project delivers via the 
Internet, ongoing OPD courses that are fully online, facilitated, and asynchronous. Each EfE 
course, delivered through a course management system, is six to seven weeks long. A typical 
session involves a selection of online readings, an activity (e.g., viewing an online video or 
creating a product), and facilitated discussion in response to a session-related prompt. This 
initiative was the subject of O’Dwyer et al.’s (2010) four randomized controlled trials as well as 
an external evaluation conducted by Boston College’s Center for the Study of Testing, 
Evaluation and Educational Policy. 


53 



Bolstering the Impact of Online Professional Development for Teachers 


Methods 

This study involved secondary analysis of teacher self-reported data from the EfE evaluation. 
The large-scale, multi-state implementation of this initiative likely affords some variability 
requisite for the explanation of variance in the outcomes of its courses. The institutional review 
board at the authors’ institution approved this study. 


Participants 

Participants were elementary and secondary teachers from nine 1 states who participated in an 
EfE course. For teachers represented multiple times in our dataset, we only include evaluation 
data from their first course. In addition, we only include teachers who voluntarily completed pre- 
, post-, and follow-up course evaluation surveys. After omitting cases with incomplete data, 
sample sizes were sufficiently large by conventions (N/k = 20): Nik = 68.4 for the model 
explaining teacher knowledge; Nik = 76.8 for the model explaining classroom practice; and N/k = 
56.0 for the model explaining student achievement. 

Instrumentation 

The data were collected immediately prior to, immediately following, and between six months 
and one year after an EFE course. Nearly all of the data were collected via surveys administered 
online between June 2006 and July 2009. Most of the evaluation survey items have Likert 
response fonnats, although two (gender and state certification status) are dichotomous. 

Some of the survey items were slightly modified for analysis. First, the response “I never needed 
it” to the item “Was technical assistance available throughout this workshop” (availability of 
technical assistance), was recoded as a central category between the alternative responses “Yes” 
and “No.” Similar modifications were made to items measuring the effectiveness of online 
discussion and participant collaboration. Originally, response options for these two items were 
on a 5-point scale [“Not applicable” (0), “Very ineffective” (1), “Ineffective” (2), “Effective” (3) 
and “Very effective” (4)]. For analysis, “Not applicable” responses were re-coded as a central 
category (i.e. between “Ineffective” and “Effective”). These modifications reasonably assumed 
that participants responding “I never needed it” or “Not applicable” had a neutral opinion 
regarding these factors. Next, gender was dummy coded such that zero and one represented 
“Male” and “Female,” respectively. Table 1 summarizes all intact (or slightly modified) 
evaluation items utilized for this study. 


1 Teachers from a tenth state did not participate in the evaluation. 
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Table 1 


Summary of Single Evaluation Items Submitted to Analysis 


Variable 

Evaluation item 

Response fonnat 

Improvement in classroom practice 

The workshop enabled me to improve my classroom 
instruction 

Agreement 

Prior teacher knowledge 

Based on the workshop description, how 
knowledgeable are you in the workshop content? 

Not knowledgeable = 1; 

Somewhat knowledgeable = 
2; Knowledgeable = 3; Very 
knowledgeable = 4 

Gender 

Do you identify as 

Male = 0; Female = 1 

State certification status 

Do you hold a state certification for your current work 
title? 

No = 0; Yes = 1 

School support 

My school supports the use of this workshop content 
in my classroom instruction 

Agreement 

Providing helpful feedback 

How successful was the facilitator at [providing 
helpful feedback]? 

Effectiveness 

Keeping discussions on-topic 

How successful was the facilitator at [keeping 
discussions on-topic]? 

Effectiveness 

Fostering stimulating discussion 

How successful was the facilitator at [fostering 
stimulating discussion]? 

Effectiveness 

Encouraging active participation 

How successful was the facilitator at [encouraging 
active participation]? 

Effectiveness 

Session reading effectiveness 

Rate the effectiveness of [session readings] in helping 
you learn the workshop material. 

Very ineffective = 0; 

Ineffective = 1; Not 

Online discussion effectiveness 

Creating a usable 

project 

effectiveness 

Participant collaboration 
effectiveness 

Rate the effectiveness of [online discussion] in 
helping you learn the workshop material. 

Rate the effectiveness of [creating a usable product by 
the end of the workshop] in helping you learn the 
workshop material. 

Rate the effectiveness of [collaborating with other 
participants] in helping you learn the workshop 
material. 

applicable = 2; Effective = 3; 
Very effective = 4 

Facilitator knowledgeableness 3 

The facilitator was knowledgeable in this content area 

Agreement 

Discussion topic relevance 3 

The topics chosen for discussion were relevant 

Agreement 

Discussion topic beneficence 3 

The topics chosen for discussion were beneficial 

Agreement 

Setting a welcoming tone 3 

How successful was the facilitator at [setting a 
welcoming tone for the workshop]? 

Effectiveness 

Clearly communicating 
expectations 3 

How successful was the facilitator at [clearly 
communicating expectations for activities]? 

Effectiveness 

Being accessible for support 3 

How successful was the facilitator at [being accessible 
for support]? 

Effectiveness 

Ease of content transferability 3 

The content of this workshop is easily transferable to 
the classroom. 

Agreement 

Organization 3 

The workshop was well-organized 

Agreement 

Interface user-friendliness 3 

The workshop website was user-friendly 

Agreement 

Culturally unbiased materials 3 

The workshop materials were culturally unbiased 

Agreement 

Availability of technical 

Was technical assistance available throughout this 

No = 0; I never needed it = 1; 

assistance 3 

workshop? 

Yes = 2 

Adequacy of compensation 3 

I received adequate credit or compensation for taking 
this workshop 

Agreement 

Linking pedagogical skills with content 
effectiveness 3 

The workshop effectively linked pedagogical skills 
and content 

Agreement 


Note. Agreement = 4-point scale: “Strongly disagree” (1), “Disagree” (2), “Agree” (3), and “Strongly agree” (4); Effectiveness = 
4-point scale: “Very unsuccessful” (1), “Unsuccessful” (2), “Successful” (3), and “Very successful” (4). 
a This variable was included in the final, exploratory regressor block. 
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In addition to employing single survey items, this study also constructed five multi-item 
composites: (a) teacher knowledge, (b) improved student achievement, (c) coherence, (d) clarity 
of goals and expectations, and (e) computer proficiency (see Table 2). Principal component and 
reliability analyses provided evidence of the unidimensionality and internal consistency of each 
composite variable (see Table 2). With the exception of the reliability of the two -item teacher 
knowledge composite (a = .52), these analyses provided evidence for the adequacy of the 
composites. All composites were the mean value of their constituent items. 

Table 2 


Summary of Composite Measures and Scale Analyses 


Composite 

Reliability analysis 

Principal components analysis 


a 

Eigenvalue for first factor 

Percent of 

explained 

variance 

Teacher knowledge outcome 

.52 

1.35 

67.50 

Improved student achievement 

.86 

2.36 

78.57 

Coherence 

.86 

1.76 

87.89 

Clarity of goals and expectations 

.93 

1.86 

93.01 

Computer proficiency 

.93 

5.46 

68.26 


We constructed the teacher knowledge outcome from two parallel items administered on the 
post-course and follow-up surveys. We employ this composite learning outcome both to improve 
the stability of the measure and capture OPD’s long-range impact. The two items were: “Having 
completed the workshop, how knowledgeable are you about the content presented in this 
workshop?” and “Now that some months have passed since taking the workshop, rate how 
knowledgeable you feel about the workshop content.” The responses for the first item were: Not 
knowledgeable = 1; Somewhat knowledgeable = 2; Knowledgeable = 3; and Very 
knowledgeable = 4. The responses for the second item were: Not at all knowledgeable = 1; Not 
very knowledgeable = 2; Knowledgeable = 3; and Very knowledgeable = 4. Item factor loadings 
for the first principal component were .82 for both items. 

The improvement in student achievement composite consisted of three items administered at 
follow-up: (a) “Students showed better academic performance in the targeted content using the 
workshop contents/approaches, (b) Students performed more difficult (in-depth) work in classes 
using workshop contents/approaches,” and (c) “Student products were of a higher quality in 
classes using the workshop contents/approaches.” The responses for these items were: Strongly 
disagree = 1; Disagree = 2; Agree = 3; and Strongly agree = 4. Item factor loadings for the first 
principal component were .85, .90, and .91, respectively. 

The coherence variable was a composite of two posttest items: “This workshop was aligned with 
my school’s professional development needs or plans,” and “This workshop addressed areas of 
curricular and/or pedagogical need in my school or district.” The responses for these items were 
the same as those for the student achievement items. Item factor loadings for the first principal 
component were .94 for both items. 
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The clarity of goals and expectations variable was a composite of two posttest items: “The 
workshop goals were clearly stated,” and “The expectations for workshop participation were 
clear.” The responses for these items were the same as those for the student achievement and 
coherence items. Item factor loadings for the first principal component were .97 for both items. 

The computer proficiency variable was a composite of eight pretest items. Participants were 
asked to respond to the question, “How proficient are you at performing each of the following” 
for (a) “Navigating websites,” (b) “Performing an Internet or library search for educational 
resources,” (c) “Downloading documents,” (d) “Uploading documents,” (e) “Reading a threaded 
discussion,” (f) “Posting comments to a threaded discussion,” (g)“Installing support programs 
(e.g., QuickTime, RealPlayer, Flash, Java, etc.),” and (h) “Troubleshooting computer programs.” 
The responses for these items were: I don’t know yet = 0; Not proficient = 1; Somewhat 
proficient = 2; Proficient = 3; and Highly proficient = 4. Item factor loadings for the first factor 
ranged from .76 to .88. 

One variable — course content — was created by the evaluation team. This variable originally 
included: technology, pedagogy, special populations, assessment and standards, and four subject 
matter areas (English language arts, mathematics, social studies and science). From this 
categorical variable, we created a dummy variable representing subject matter focus, by setting 
the four subject matter areas equal to 1 and all others equal to 0. 

Analytic Approach 

We conducted separate blockwise ordinary least squares (OLS) regression analyses to explain 
variation in each of our criterion variables (increased knowledge, improved classroom practice, 
and improved student achievement). The OLS modeling is identical for all three outcomes. The 
only exception is that we include the teacher knowledge outcome as a predictor of improved 
classroom practice, and both the teacher knowledge and improvement in classroom practices 
“outcomes” as predictors of improved student achievement. We chose this modeling approach 
because these variables constitute mediators in our professional development theory of action 
(see Figure 1). Due to the number of significance tests, we selected a significance level of .01. 


Online 

Professional 

Development 


Teacher 

Knowledge 


Classroom 

Practice 

* 

Student 

Achievement 


Figure 1 . Conceptual model depicting the relationship between online professional development 
activities and its intermediate and ultimate outcomes. 
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For all three outcomes, the first block includes teacher and school contextual characteristics 
(standard entry). Specifically, the model includes prior knowledge, gender, state certification 
status, computer proficiency, and school support. The inclusion of prior knowledge in course 
content, gender, and state certification for one’s current teaching position (a proxy for teacher 
training) is intended to partial out extraneous variance. The mediators (teacher knowledge and 
improvement in classroom practices) are also included in this first block for the classroom 
practice and student achievement regressions. The second regressor block includes two 
established characteristics of effective professional development, content focus and coherence 
(also standard entry). The third block contains OPD learner activity variables (analogous to 
active learning opportunities); this block allowed for the identification of OPD course 
“activities” that are related to its impact (again standard entry). The second and third blocks are 
intended to replicate what is known about traditional PD but apply them to an online context. 
The fourth block includes previously uninvestigated factors afforded by our data source (forward 
stepwise entry with p < .01 as the statistical criterion for entry). 

Results 

Table 3 presents descriptive statistics for all variables. Inspection of variance inflation factor 
values did not suggest major multicollinearity issues (VIFs < 5), though there is multicollinearity 
in these data. While the regression coefficients may be affected somewhat by this 
multicollinearity, we chose not to remove variables from the models. All reported coefficients 
are standardized and from the final model (see Table 4). 

Our final regression model for teacher knowledge explained 32% of the variance ( R A ) (see Table 
4). The block of individual and school contextual variables explained 17% of the variance (A R"). 
All variables within this block except for gender were statistically significant. As expected, 
participants who self-reported higher levels of knowledge in course content at the outset reported 
more knowledge after the OPD (P = .13). By including prior knowledge in the model, we have 
attempted to partial out variation in the outcome that is “due to” this predictor. We henceforth 
interpret the relationships between other predictors and the criterion in terms of change in 
knowledge. Increases in the computer proficiency composite (P = .18), and school support for 
instructional use of course content (p = .16) were both also associated with gains during the 
course. Not having a state certification for one’s current teaching position (P = -.08) was also 
associated with significantly more gains. 

The second and third blocks — those representing established predictors of PD effectiveness — 
respectively explained 5.7% and 6.3% additional variance in the teacher knowledge outcome. 
Within the second block, only coherence was statistically significant (p = .09). Increases in 
coherence were associated with gains in teacher knowledge. In the third block, only one of the 
eight learner activity variables was statistically significant: the effectiveness of session readings 
(p = .08). Increases in the rated effectiveness of this online course element were associated with 
teacher knowledge gains. 
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Table 3 


Descriptive Statistics for All Variables 



Teacher 

knowledge 

(N = 1231) 

Classroom 

practice 

(N = 1228) 

Student 
achievement 
(N= 1008) 

Measure 

M 

SD 

M 

SD 

M 

SD 

Teacher knowledge outcome 

3.37 

.45 

3.37 

.45 

3.41 

.44 

Improvement in classroom practice 

- 

- 

3.33 

.60 

3.41 

.56 

Improvement in student achievement 

- 

- 

- 

- 

3.06 

.49 

Prior teacher knowledge 

2.24 

.71 

2.24 

.71 

2.25 

.71 

Gender 

.89 

.32 

.89 

.32 

.89 

.31 

State certification status 

.94 

.23 

.94 

.24 

.95 

.22 

Computer proficiency 

2.66 

.88 

2.67 

.88 

2.67 

.86 

School support 

3.34 

.59 

3.34 

.59 

3.40 

.57 

Content focus 

.52 

.50 

.52 

.50 

.53 

.50 

Coherence 

3.58 

.49 

3.58 

.49 

3.61 

.48 

Providing helpful feedback 

3.70 

.52 

3.70 

.52 

3.72 

.50 

Keeping discussions on-topic 

3.68 

.51 

3.68 

.51 

3.70 

.50 

Fostering stimulating discussion 

3.63 

.55 

3.63 

.55 

3.65 

.54 

Encouraging active participation 

3.71 

.48 

3.71 

.48 

3.73 

.47 

Session reading effectiveness 

3.43 

.56 

3.43 

.56 

3.45 

.55 

Online discussion effectiveness 

3.44 

.69 

3.44 

.69 

3.46 

.66 

Creating a usable project effectiveness 

3.49 

.63 

3.48 

.63 

3.51 

.60 

Participant collaboration effectiveness 

3.31 

.79 

3.31 

.79 

3.33 

.76 

Facilitator knowledgeableness 

3.74 

.45 

3.74 

.45 

3.75 

.45 

Discussion topic relevance 

3.75 

.45 

3.75 

.45 

3.76 

.44 

Discussion topic beneficence 

3.74 

.47 

3.74 

.47 

3.76 

.46 

Setting a welcoming tone 

3.83 

.40 

3.83 

.40 

3.84 

.38 

Clearly communicating expectations 

3.74 

.48 

3.74 

.48 

3.75 

.47 

Being accessible for support 

3.75 

.48 

3.75 

.48 

3.76 

.47 

Ease of content transferability 

3.66 

.51 

3.66 

.51 

3.69 

.49 

Clarity of goals and expectations 

3.71 

.48 

3.70 

.48 

3.72 

.47 

Organization 

3.74 

.46 

3.74 

.46 

3.75 

.46 

Interface user-friendliness 

3.68 

.51 

3.68 

.51 

3.69 

.51 

Culturally unbiased materials 

3.72 

.47 

3.72 

.47 

3.73 

.46 

Availability of technical assistance 

1.48 

.54 

1.48 

.54 

1.49 

.54 

Adequacy of compensation 

3.49 

.59 

3.49 

.59 

3.51 

.58 

Linking pedagogical skills with content 

3.55 

.52 

3.55 

.52 

3.57 

.52 
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The stepwise procedure entered one additional variable into the teacher knowledge model 
(explaining an additional 3.1% of the variance). In particular, the extent to which the course 
content was easily transferable to the classroom (P = .21) was associated with teacher knowledge 
gains. The absolute magnitudes of the final teacher knowledge model’s standardized regression 
coefficients (Ps) were generally small (.08-.21). In this final model, the variables most related to 
the criterion were the ease of content transferability (.21), computer proficiency (.18), and school 
support (.16). 

Our final OLS regression model for improved classroom instruction explained 42% of the 
criterion variance ( R \ ) (see Table 4). The first block of individual and school contextual 
variables accounted for nearly all of this variance (41.6%). However, only two predictors in this 
block, the teacher knowledge mediator (p = .22) and school support for instructional use of 
course content (P = .51), were statistically significant. Increased self-reported knowledge after 
the OPD course was associated with improvement in self-reported classroom practice. School 
support for instructional use of course content was also associated with improvements in 
classroom instruction; the magnitude of this relationship (i.e., p = .51) is notable. None of the 
variables in the other three blocks were significant. 

Our final OLS regression model for improved student achievement explained 20% of the 
criterion variance ( R 2 a ) (see Table 4). The individual and school contextual variables explained 
16.6% of the variance. However, only the improved classroom practice mediator was statistically 
significant (p = .31), indicating improved classroom practice was related to improved teacher- 
reported student achievement. None of the variables in the second and third blocks were 
statistically significant at .01. The stepwise procedure added one variable to the model, the 
availability of technical assistance, which explained a very small additional amount of variance 
(.7%). Increases in the availability of technical assistance were associated with increases in 
student achievement (p = .09). 
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Table 4 


Summary of Blockwise Linear Multiple Regression Analysis 


Predictors 

Teacher 

Classroom 

Student 


knowledge 

practice 

achievement 

Block 1 

Prior knowledge 

13*** 

-.03 

-.02 

Gender 

-.00 

.03 

.03 

State certification 

-.08** 

.03 

.01 

Computer proficiency 

lg*** 

.01 

.05 

School support 


52*** 

.01 

Teacher knowledge 

- 

22*** 

.06 

Classroom practice 

- 

- 

31*** 

Block 2 

Content focus 

-.01 

.01 

-.06 

Coherence 

.09** 

.02 

.07 

Block 3 

Providing helpful feedback 

.05 

-.02 

.00 

Keeping discussions on-topic 

.05 

.01 

-.12 

Fostering stimulating 
discussions 

-.06 

.02 

.14 

Encouraging active 
participation 

.01 

.04 

.00 

Session readings 

.08** 

.02 

.03 

Online discussions 

.06 

.03 

.01 

Creating a usable project 

.06 

.02 

.08 

Participant collaboration 

.06 

-.00 

.02 

Block 4 

Ease of content 
transferability 


- 

- 

Availability of technical 
assistance 

- 

- 

.09** 

Model F 

36.35*** 

56.26*** 

14.51*** 

R 2 

.32 

.43 

.21 

Ra 

.32 

.42 

.20 


Note. All reported regression coefficients are standardized. **p < .01. ***p < .001. 
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Discussion 

With this study, we have attempted to explain variation in OPD’s intermediate and ultimate 
outcomes. We find clear evidence for the validity of the teacher-practice-student achievement 
PD conceptual framework (see Figure 1). Consistent with earlier studies by Garet, Porter, 
Desimone, Birman and Yoon (2001) and Ingvarson, Meiers and Beavis (2005), teacher 
knowledge predicts improvements in classroom practice, which in turn predicts improvements in 
student outcomes. It would seem that this series of events must be in place in order to advance 
student achievement outcomes with OPD or PD. 

For teacher knowledge, the statistical significance of prior knowledge and state certification 
status underscores the importance of accounting for background knowledge and partialing out 
extraneous variance in an investigation such as this. Regarding the state certification finding, 
Garet et al. (2001) did find a similar inverse relationship between having an in-field certification 
and changes in their outcome, teaching practices. This might suggest that the out-of-field 
teachers stand to gain more from OPD (or any PD for that matter) than their certified 
counterparts. However, supplemental analyses showed that in- and out-of-field teachers did not 
differ significantly in terms of their pre-course knowledge. 

More importantly, we replicate findings that participants’ computer proficiency [discussed by 
Zhao et al. (2005) and Tallent-Runnels et al. (2006)], school support (Ingvarson et al., 2005), and 
coherence (Garet et al., 2001) are related to teacher knowledge gains. The first finding clearly 
suggests that developers of OPD should be mindful of participants’ computer proficiency. Our 
interpretations of the school support and coherence findings center on teachers’ motivations. It is 
possible that teachers’ perceptions of school support for implementing PD content may bear on 
their motivation to learn that content. Teachers may similarly be more motivated to leam PD that 
is coherent, or aligned, with local professional development initiatives and needs. 

Next, our learner activity block showed that the rated effectiveness of session readings was 
related positively to the amount of knowledge growth reported during the course. To the best of 
our knowledge, this is the first study evidencing a mechanism by which OPD may be producing 
effects and also that the quality of this mechanism does indeed matter. Not surprisingly, online 
course quality matters (see Tallent-Runnels et al., 2006). 

In addition, we find evidence that more teacher learning takes place when OPD content can be 
transferred easily to a classroom setting. We suspect that this finding indicates that teachers leam 
more (or are perhaps motivated to learn more) in OPD that has practical, concrete, and readily 
usable, rather than theoretical, content (Darling-Hammond & McLaughlin, 1995; Guskey, 2002). 
In an earlier OPD study, Reeves and Pedulla (2011) found that teachers were more satisfied with 
such easily transferable content. Moreover, this finding is consistent with the adult learning 
literature, which suggests that this population desires knowledge that can be applied immediately 
(Cercone, 2008). 
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On the other hand, we do not find that a focus on content (i.e. math, language arts, etc.) is 
associated with increases in teacher knowledge. We offer the following explanation for the 
incongruence of this finding and the PD effectiveness literature: It is worth keeping in mind that 
typical investigations of PD effectiveness focus on PD that is implemented by schools or 
districts; it is for this form of PD that a focus on content has been shown to be most effective. In 
an OPD context, where teachers voluntarily enroll, it is possible that teachers may be motivated 
to learn the course material, whatever its content might be. 

For improvements in classroom practice, two statistically significant variables accounted for 
42% of the variance. First, we find a statistically significant effect of the teacher knowledge 
mediator. Second, school support for instructional use of workshop content was also significantly 
related to improvements in classroom practice. The fact that this variable is statistically 
significant for this particular outcome is understandable, as it captures the context in which 
teachers actually engage in their classroom practice. School support is, as Ingvarson et al. (2005) 
suggested, an “an important enabling condition” (p. 15) for curricular implementation of content 
and pedagogical change. In addition, while school support was also significant for the teacher 
knowledge analysis, the magnitude of the effect is larger for classroom practice, suggesting that 
school support is particularly important for classroom implementation. This finding, more than 
any other in this study, exemplifies the utility of linking particular design/implementation 
features to particular outcomes when studying the effectiveness of OPD. For classroom practice, 
we also do not find evidence for an effect of content focus, a result for which we offered an 
explanation earlier. We also do not find an effect of coherence on classroom practice, although 
this factor was statistically related to changes in teacher knowledge. 

Our only finding for the student achievement analysis was that the availability of technical 
support was associated with improved student outcomes. This finding converges with a finding 
by Penuel, Fishman, Yamaguchi and Gallagher, (2007) that the provision of technical support to 
teachers was predictive of PD content implementation. A need for technical support is not 
surprising given evidence for non-stellar computer proficiencies among some teachers (Reeves 
& Li, 2012). 

This study’s findings must be interpreted in light of its limitations, which include possible 
selection biases, and more crucially its data and design. Limitations such as these are quite 
characteristic of observational studies, including much of the published literature cited 
throughout this article. Given these potential threats, however, a number of things are worth 
noting. In light of the possible selection bias, we restricted our target population to only teachers 
who chose to enroll in an OPD course. With regard to internal validity, our study did attempt to 
exercise control of extraneous variables that might confound observed relationships. Because of 
our study’s correlational design, we present here only functional rather than bona fide causal 
relations. Next, despite some measurement limitations, each of the outcome variables were 
measured, in part, at least six months after the course, which is a strength of the measurement. 
The fact that data were self-reported is also a limitation. Since research along these lines was 


63 



Bolstering the Impact of Online Professional Development for Teachers 


needed, we believe that the evidence reported here represents an important first step toward 
understanding some of the variables that can maximize the impact of OPD. 

Nevertheless, future research should surely attempt to replicate findings reported here using 
samples of teachers in different OPD contexts and instrumentation designed specifically to 
measure the variables of interest. More broadly, further research is needed on how to measure 
effectiveness in and evaluate OPD (Dede et ah, 2009). Research utilizing data collected within 
and by course management systems should also prove informative. 

Barring these limitations and the need for additional research, this study responds to calls in the 
literature for research on the effectiveness of online professional development (e.g., Ginsberg et 
ah, 2004). Taken altogether, these findings suggest a number of factors that are related to OPD’s 
intennediate and ultimate impacts. Given earlier research, coupled with the findings presented 
here, we offer the following implications: Clearly, designers and implemented, as well as 
facilitators, of OPD should be mindful of teachers’ computer proficiency. As a potential solution 
to this issue, a built-in or stand-alone training module for the relevant skill set could be made 
available to participants. Next, school support for the use of knowledge garnered through OPD 
appears important, having shown up as a predictor of both teacher knowledge and improved 
classroom practice. This finding implies that it might behoove administrators to endorse openly 
the curricular implementation of OPD or PD content. Regarding the in-service training itself, 
OPD that is coherent with professional development needs and initiatives in teachers’ schools 
and features high-quality readings can be expected to yield a greater impact. This study also 
suggests a benefit to teacher learning when the content is practical, concrete and readily usable, 
and relevant to teachers’ day-to-day professional needs. Finally, making available technical 
support during OPD may prove beneficial. 

In conclusion, given that dissemination was an explicit part of Boston College’s role in the e- 
Learning for Educators initiative, this study shares empirical lessons learned for others hoping to 
design and implement facilitated OPD for teachers. The study reported here represents a much- 
needed move toward effective professional development theory and model building (Dede et ah, 
2009; Desimone, 2009). It is clear that high-quality professional development, including that 
offered online, can help in tackling school reform and meeting the ambitious standards for 
student achievement called for in the No Child Left Behind Act 2001 (20 U.S.C. § 6319 (2008), 
Borko, 2004) and that will no doubt continue with its reauthorization in the coming year. 
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